With the growing influence of AI in our day to day life, integrating of LLM models/AI in enterprise applications will be going to be an essential/critical part. AI will not be a feature going forward, it will be the essential part for any enterprise application which will help users by taking decisions and helping them to fulfill their task.
So it means eventually enterprises will need to move their core applications to Python for integrating AI in their software/application?
The answer is NO, Python dominates the AI world that is true but with the evolution of frameworks such as LangChain4j gives developers full power of LLMs orchestration in Java ecosystem.
In this article we will explore our experience of integrating LangChain4j with locally run Ollama models through Java frameworks such as Spring Boot, Quarkus etc. This will help us in building enterprise enabled agentic apps.
What is LangChain4j?
Langchain4j provides the framework to integrate LLMs modes in the Java related applications. It hides the complexity of integrating with the LLM, so that all the boilerplate code will be abstracted through the structured interfaces. LangChain4j is inspired by Langchain and LLamaindex which are pure Python frameworks.
It helps Java developers power to integrate with different LLMs, vector databases, embeddings and tools/agents. With these frameworks Developers need to focus only on the business logic and not worry about the boilerplate code of integrating with LLMs, vector databases etc. Going forward any change in the LLMs, databases, embeddings will need the configuration change without impacting their core logic.
LangChain4j exposes different BOMs/jars to integrate with Java Frameworks such as Spring Boot/ Quarkus etc. Since in the current scenarios most of the enterprise software is built on Java and its related frameworks, Langchain4j helps enterprises to enhance their products/apps with power of agentic systems without changing their existing technology.
With the power of Java and LangChain4j exposes the strongly typed prompts, responses and tool signature. Plug in the product graded observability into your existing tracing, logging and metrics etc.
Let’s start the article by exposing your First Agentic Services.
For a spring Boot 4 project, to interact with Ollama needs to add the following dependencies in your pom.xml
<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-spring-boot4-starter</artifactId>
<version>1.14.1-beta24</version>
</dependency>
<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-ollama-spring-boot4-starter</artifactId>
<version>1.14.1-beta24</version>
</dependency>If you are using OpenAI then need to add the OpenAI specific dependency. For Spring Boot:
<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-open-ai-spring-boot-starter</artifactId>
</dependency>LangChain4j provides integration with different models of
● Anthropic
● OpenAI
● Ollama
● Gemini
● Hugging-face etc.
For more Details check the official Git document of LangChain4j
2- Configure model in Langchain4j
For configuring your models, you need to add the following configuration in your application.
For Spring boot add following configuration in your application.yaml/appplication.properties:
langchain4j.ollama.chat-model.base-url=http://localhost:11434
langchain4j.ollama.chat-model.model-name=llama3.2
langchain4j.ollama.chat-model.log-requests=true
langchain4j.ollama.chat-model.log-responses=true
langchain4j.ollama.chat-model.temperature=0.0The above configuration is very specific to Ollama which will be running on your local system, where we need to give the basic url, model name, logging information and temperature.
Temperature tells about how creative you want your LLM to perform, with 0.0 value you want your system to give a more deterministic response. Higher the value means the output will be more creative.
If you want to use open-ai or other LLMS then change the configuration accordingly.
3-Create your first AgenticService
We will use the power of LangChain4j to define the AgenticService, by defining what needs to be done and where part of interacting with LLM will be taken care of by LangChain4j.
@AiService
public interface TravelAgentService
{
String travelAgentChat(@UserMessage String userMessage);
}Now use the power of spring boot to inject this in your controller and take the prompt from the user and pass it to the LLM through agentic Service.
@RestController
@RequestMapping("/api/travelagent")
public class ChatApi
{
@Autowired
private TravelAgentService travelAgentService;
@GetMapping("/chat")
public ResponseEntity<String> chat(@RequestParam String question) {
return ResponseEntity.ok(travelAgentService.travelAgentChat(question));
}
}Response from service
4- Maintaining Session in LangChain4j
LLMs and our rest api’s are stateless and they don’t remember previous query/question. But for enterprise applications storing the user chat history is a critical thing. For that LangChain4j provide Session to maintain foreach Session
In Spring Boot first need to configure the session configuration
| @Configuration public class SessionConfiguration { @Bean public ChatMemoryProvider sessionProvider() { return memoryId -> MessageWindowChatMemory.builder() .id(memoryId) .maxMessages(20) // store last 20 messages .build(); } } |
Then use this within your Agentic Service
| @AiService (chatMemoryProvider = “chatMemoryProvider”) public interface TravelAgentService { String travelAgentChat(@MemoryId String sessionId, @UserMessage String userMessage); } |
Now my Agentic Service will keep the session information persisted for 20 messages against this session id.
Let’s understand how this works
First Input from user 1
– http://localhost:8080/api/travelagent/chat?sessionId=”1″&question=”My query is for India Famous destination”
[Application Response] :
| India has numerous famous destinations that attract millions of tourists every year. Here are some of the most popular ones: 1. **Taj Mahal**: Located in Agra, Uttar Pradesh, the Taj Mahal is one of the Seven Wonders of the World and a symbol of love. |
First Input from user 2:
– http://localhost:8080/api/travelagent/chat?sessionId=”2″&question=”What is my Last Query about?”
[Application Response] :
| You didn’t ask a question, so you haven’t made a query yet. This conversation just started. What would you like to talk about? |
Second Input from user 1
– http://localhost:8080/api/travelagent/chat?sessionId=”1″&question=”What is my Last Query about?”
[Application Response] :
| Your last query was “My query is for India Famous destination”. You were asking me to suggest some of the most famous destinations in India. |
This is how the session id works in the LangChain4j. LangCahin4j also provide the TokenWindowChatMemory to keep the chat open for limited Tokens only.
| TokenWindowChatMemory.builder() .id(memoryId) .maxTokens(1000, new SimpleTokenCountEstimator()) .build(); |
You can also store the chat history in your RedisCache so that it survives, if the user session got restarted.
MessageWindowChatMemory.builder() .id(memoryId) .maxMessages(20) .chatMemoryStore(redisChatMemoryStore())
While giving the response to the user, there may be different actions needed to complete a task such as querying Database or doing the web search or invoking the other apis to get more context etc. For this LangChain4j provides tools or functional calls to do these extra work which LLMs are not capable of.
LLMs invoke these tools on their own depending upon the prompts given to them as system messages. You can invoke multiple tools for a single agentic service depending on your business logic.
In Spring Boot add the following tool to add today’s date in the response
| @Tool(“”” Get today’s date. Call this tool before giving the response to get today’s date. Add this date in the response. “””) public String getDate() { return LocalDate.now().toString(); } |
This is how tools are defined in the spring boot application, description added should be so descriptive that LLM understands what this tool is used for and when to invoke this tool.
| @AiService(tools = “AgenticTools.class”) public interface TravelAgentService { @SystemMessage(“”” You are a helpful agent. Follow these steps IN ORDER for EVERY response. DO NOT SKIP ANY STEP: Step 1: MANDATORY – Call getDate tool RIGHT NOW before anything else. Do not proceed to Step 2 without calling getDate first. Step 2: Prepare a COMPLETE and DETAILED answer to the user’s question as plain text. RULES: – Step 1 is MANDATORY tool calls — never skip it – Always call getDate FIRST before preparing any answer – Never assume or guess the date — always get it from getDate tool “””) String travelAgentChat(@UserMessage String userMessage); } |
While calling the LLM in AgenticService add the system message, where giving the instruction to the LLM that how the tool is invoked and where to add tool response. Now we can see the response of the Service will contain today’s date
This is the power of Tools, tools can do other task also for example sending response to external Services
Taking Example where whatever response is generated by LLM we need to pass the same response to the External API which is on WebHook before displaying to the user.
| @Tool(“MANDATORY: Always call this tool with your complete answer before responding to user.”) public String sendWebhook(@P(“Complete answer text as plain string” ) String message) { try { System.out.println(“tested message::”+ message); String webhookUrl = “https://webhook.site/<WEB_HOOK_TOKEN>”; //Replace the token with your actual token Map<String, String> payload = new HashMap<>(); payload.put(“message”, message); payload.put(“timestamp”, LocalDateTime.now().toString()); HttpHeaders headers = new HttpHeaders(); headers.setContentType(MediaType.APPLICATION_JSON); String jsonBody = objectMapper.writeValueAsString(payload); HttpEntity<String> request = new HttpEntity<>(jsonBody, headers); restTemplate.postForObject(webhookUrl, request, String.class); return “WEBHOOK_SENT_SUCCESSFULLY. Do not call sendWebhook again. Return answer to user now.”; } catch (Exception e) { return “Error: ” + e.getMessage(); } } |
Use this tool with the prompts in the Agentic Service
| @SystemMessage(“”” You are a helpful agent. STRICT WORKFLOW – follow exactly: Step 1: Read the user question. Step 2: Generate a complete detailed answer to the question. Step 3: Call sendWebhook with YOUR ANSWER from Step 2. NOT the user question. NOT a summary. Your full detailed answer. Step 4: Return your answer to the user. RULES: – Step 3 is MANDATORY tool calls — never skip it – Always call sendWebhook tool – Never assume or guess the date — always get it from getDate tool “””) String travelAgentChatAndWebHook(@UserMessage String userMessage); |
Now when this API will be invoked we will see the response is printed to the user and also in the WebHook site.
Response to the User:
Response from the WebHook Site:
Tools are a critical part of LangChain4j, and should be used very carefully for deterministic responses. System messages should contain the instructions properly to get the optimal response otherwise your responses can be hallucinated.
Tools can contain Skills or MCP for LLM to make the quick decision while performing any task.
The last topic for this article is Integrating the MCP.
6- Integrating MCP using Agentic Tools.
Tools can be used to integrate your MCP functions or calls which will be needed to complete your business task.
In this example we will create a Java based MCP stdio server (a local subprocess launched by an MCP client), you can also invoke the MCP through url.
For this first create the StdioMcpTransport where we need to provide the command that runs on your local system. Wrap this transport layer to the DefaultMCPClient/
| @Bean public McpClient mcpClient() { StdioMcpTransport transport = new StdioMcpTransport.Builder() .command(List.of( “npx.cmd”, “-y”, “@modelcontextprotocol/server-filesystem”, “.” )).logEvents(true) .build(); return DefaultMcpClient.builder() .transport(transport) .build(); } |
Then wrap this client inside the MCPToolProvider which will be used by the Agentic Service.
The above mentioned command is to search the filesystem using npx utility.
Make sure you have npx utility installed in your system.
| @Bean public ToolProvider mcpToolProvider(McpClient mcpClient) { return McpToolProvider.builder() .mcpClients(mcpClient) .build(); } |
| @AiService(toolProvider = “mcpToolProvider”) public interface McpAiAgenticService { @SystemMessage(“”” You are a helpful agent with access to filesystem tools. Use available tools to answer questions. Always use tools when needed. “””) String chat( @UserMessage String userMessage); } |
LangChain4j also helps to create the MCP Server and MCP client, which can communicate with each other.
LangChain4j is not limited to the above mentioned tools or functionality. These are core functionality which will be the starting point for learning the LangChain4j. LangChain4j also provides the integration with various vector Databases, Embedding Stores and GuardRails.
LangChain4j provides the integration with Observability, it emits events at every stage of the pipeline. Hook into them with a `ChatModelListener` and forward to your metrics system:
Bigger Picture as an Architecture
The shift happening right now isn’t about adding a chatbot to your app. It’s about AI becoming an integral component — sitting alongside your database, your message broker, your cache.
LangChain4j helps Java developers to do the following shift:
– RAG pipelines: Langchain4j helps to replace the bitter keyword search with semantics understanding.
– MCP Servers and Clients: Langchain4j will be used for creating the MCP servers and MCP clients and also helps to communicate through Tools.
– Streaming Response: Giving the partial response while LLM is doing the processing behind the scenes
– Structured Response: You can give the Structured response of JSON instead of giving the String response back to the user.
– Token management: You can also manage the tokens that are being consumed so far, putting a limit to these Token consumption for one session etc.
The Java ecosystem has spent decades building patterns of reliability, type safety, scale etc. LangChain4j extends these patterns into the AI domain.
Points to be noted:
– LangChain4j is still in the maturing phase and hence it might be possible that some features will be available in LangChain and not in LangChain4j.
– Ollama provides different results with different models, use the one which is more suitable for your business needs.
– Ollama response time is more as compared to the paid LLMs which take less time and give more deterministic response.
– Response added here may vary when you run in local and also the system messages need to be changed for other LLMs to give the deterministic response.
– For detailed code examples of this article, please refer to my Git repository.
